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UNRELIABLE GUIDELINES 


RELIABLE SOURCES AND MARGINALIZED 
COMMUNITIES IN FRENCH, ENGLISH AND 
SPANISH WIKIPEDIAS 


Report of the research project Reading Together: Reliability 
and Multilingual Global Communities 


Executive Summary 


In ahistorical moment typified by concerns about the proliferation of propaganda and 
misinformation online, we askto what extent are contributions from and content about 
marginalized communities affected by guidelines about reliability in three language 
versions of the world's largest crowdsourced online encyclopedia? This report takes 
on this question and addresses how Wikipedia trainers involved in the Art Feminism 
movement approach the reliable source guidelines in French, English and Spanish 
Wikipedias. Based on community conversations and interpretative analysis of the 
guidelines, the report shows that on Wikipedia source authority is facilitated by social 
and technical processes which elevate the decisions of a small number of self- selected 
editors. Trainers play an essential role in translating guidelines and processes, and take 
on additional labor in managing editorial friction. We also highlight that translations of 
texts about reliability between different language versions of Wikipedia tend to center 
Western knowledge formations. This report describes our methodology; our findings; 
and the effects for marginalized communities and readers writ large. Reliability should 
not mean exclusion: we end with recommendations for enacting more inclusive and 
diverse Wikipedias. 
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M DIRECTIVES PEU FIABLES 


Le LES SOURCES FIABLES ET LES COMMUNAUTÉS 
p MARGINALISÉES DANS LES WIKIPÉDIAS 
e ANGLAISE, ESPAGNOLE, ET FRANCAISE 


ur Repport du projet de recherche Lire ensemble : Fiabilité et 
SES: communautés mondiales multilingues 
cof Résumé 


Dans un moment historique caractérisé par des préoccupations concernant la 
prolifération de la propagande et de la désinformation en ligne, nous posons la 
question : dans quelle mesure les contributions et le contenu des communautés 
marginalisées sont affectés par les directives sur la fiabilité dans les trois versions 
f. 2 ff linguistiques de la plus grande encyclopédie en ligne du monde ? Ce rapport aborde 
LION cette question et traite de la maniére dont les formateurs et formatrices de Wikipédia 
“C J impliqués dans le mouvement Art+Féminisme abordent les directives sur la fiabilité 
HS 2j des sources dans les versions de Wikipédia en français, anglais et espagnol. Se 
| basant sur des conversations communautaires et sur une analyse interprétative des 
t JE directives, ce rapport montre que sur Wikipédia, l'autorité de la source est facilitée par 
Ge; AS: des processus sociaux et techniques qui privilégient les décisions d'un petit nombre 
tes d'éditeurs et éditrices auto-sélectionnés. Les formateurs et formatrices jouent un róle 
TAI. essentiel dans la traduction des directives et des processus, et assument un travail 
oe 21 supplémentaire dans la gestion des frictions éditoriales. Nous soulignons également 
- fc que les traductions de textes sur la fiabilité entre les différentes versions linguistiques 
L2 220 de Wikipédia ont tendance à centrer les formations de connaissances occidentales. 
Ce rapport décrit notre méthodologie, nos résultats et les effets sur les communautés 
fe: marginalisées et les lecteurs et lectrices en général. La fiabilité ne doit pas étre 
M synonyme d'exclusion : nous terminons par des recommandations pour la mise en 
"Cg place des versions de Wikipédia plus inclusives et diversifiées. 
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Informe del proyecto de investigación Reading Together: 
Fiabilidad y comunidades globales multilingües 


Resumen 


En un momento histórico caracterizado por la preocupación por la proliferación de 
la propaganda y la desinformación en línea, nos preguntamos hasta qué punto las 
contribuciones de las comunidades marginadas y los contenidos sobre ellas se ven 
afectados por las directrices sobre fiabilidad en las versiones en tres idiomas de la 
mayor enciclopedia en línea de código abierto del mundo. Este informe aborda esta 
cuestión y se ocupa de cómo las/los/les capacitadora/es de Wikipedia que participan 
en el movimiento Arte+Feminismo abordan las directrices sobre fuentes fiables en 
las Wikipedias francesa, inglesa y espanola. Basándose en las conversaciones de la 
comunidad y en el análisis interpretativo de las directrices, el informe muestra que en 
Wikipedia la autoridad de las fuentes se ve facilitada por procesos sociales y técnicos 
que elevan las decisiones de un pequeno numero de editores autoseleccionados. 
Las/los/les formadoras/es desempenan un papel esencial en la traducción de las 
directrices y los procesos, y asumen una labor adicional en la gestión de las fricciones 
editoriales. También destacamos que las traducciones de textos sobre la fiabilidad 
entre las distintas versiones lingüísticas de Wikipediatienden a centrar las formaciones 
de conocimiento occidentales. Este informe describe nuestra metodología, nuestros 
resultados y los efectos para las comunidades marginadas y lectoras y lectoras en 
general. Lafiabilidad no deberíasignificarexclusión:terminamos con recomendaciones 
para construir Wikipedias más inclusivas y diversas. 
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INTRODUCTION 


Research on Wikipedia's guidelines and editorial processes on reliable sources matters 
in this historical moment typified by the propagation of mis and disinformation in the 
digital information ecosystem. Reading Together: Reliability and Multilingual Global 
Communities was a multilingual research project conducted from September 2020 
through March 2021. It was developed by members of Art* Feminism, an international 
community working to improve Wikipedia's content on gender, feminism, and the arts 
since 2013, and partially funded by WikiCred. 


The goal behind this effort was to understand the effects of the current set of 

reliable source guidelines and rules on the participation of and the content about 

marginalized communities on Wikipedia. We choose to work on the English, French, 

and Spanish language versions, as these are ranked among the top 10 Wikipedias in 

number of articles (1, 5 and 9, respectively.)* However, our analysis can be replicated 

in other languages. In this research, the term "marginalized" refers to communities 

and individuals who are currently underrepresented in Wikipedia, both in terms of 

participation and content. This categorization includes, but is not limited to, cis and 

transgender women, non-binary people, non-Western communities, 2SLGBTQI+ (Two- 

Spirit, lesbian, gay, bisexual, transgender, queer, questioning, 

Our findings showcase intersex), and BIPOC (Black, Indigenous, and People of Color) 

how in French, English communities. This study focused on naming what constitutes 

and Spanish Wikipedias, the reliable source guidelines on English, French, and Spanish 

Wikipedias and how source authority is negotiated among the 

Wikipedia users who do editorial work in these communities, 
and the effects for marginalized communities. 


contextualized knowledges 
that matter to marginalized 
communities are —at best— 
challenged by the definitions Our findings showcase how in French, English and Spanish 
of “reliable source.” Wikipedias, contextualized knowledges that matter to 
marginalized communities are—at best—challenged by the 
definitions of “reliable source.” Source authority is facilitated by the social and 
technical processes that elevate the decisions of a small number of self-selected 
editors. Trainers play an essential role in translating guidelines and processes, but also 
take on additional labor to manage friction between new editors and editors on-wiki. 
We also note that translations of texts about reliability between Wikipedias tend to 
center Western knowledge formations. This report describes how we conducted our 
research; our findings; and the effects for marginalized communities and readers writ 
large. Reliability should not mean exclusion: we end with recommendations for more 

inclusive and diverse Wikipedias. 


°1 Meta contributors, “List of Wikipedias." 
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BACKGROUND 


Founded in 2001, Wikipedia.org is an internet-born crowdsourced online encyclopedia 
that is governed by volunteer users and hosted by the non-profit Wikimedia Foundation 
(WMF) in the United States. The WMF supports affiliated projects and the Wikimedia 
movement to bring "free educational content to the world."? There are 310 active 
language editions of Wikipedia.? The largest have millions of articles; English Wikipedia 
has more than 6 million articles, French Wikipedia has more than 2.2 million articles, 
and Spanish has more than 1.6 million articles. Beyond article numbers, Wikipedia 
content is widely and freely accessed around the world.^ 


Like commercial digital media "giants" hosted in the United States, both the WMF and 
Wikimedia users are immunized from legal liability for the contents on the platform 
and granted the right to make governance decisions under the 1996 Communication 
Decency Act, Section 230. In other words, the Wikimedia Foundation does not make 
editorial decisions. Rather, the Wikimedia Foundation supports volunteers to manage 
the Wikimedia platforms. This governance strategy is distinct from the content creation 
and management processes of other popular for-profit platforms, such as Facebook, 
Twitter, YouTube (Alphabet) or Instagram (Facebook). Though these platforms 
encourage users to join and share “user-generated content,”° platform governance— 
including what and why content is featured, prioritized, or minimized—is undertaken 
by the company. Social theorists have advanced concepts such as the commercial, 
networked "attention economy" and “super public,” “surveillance capitalism" ^ and 
"affective publics”? which name ways that the neoliberal ideologies and emotional 
networks animate page views, sharing, and visibility. 


However, these concepts are less useful to explain the dynamics of user interaction 
and governance on Wikipedia, due to the peculiarities of its user governance. In 
the popular press, Wikipedia has increasingly been described in opposition to the 
commercial platforms as an internet success story.? English Wikipedia's achievements 
as a user-governed platform have gained appreciation in light of the proliferation of 
misinformation and disinformation on social media platforms. English Wikipedia, 
largely, has relied on core content strategies and community policies and arguably 


02 "Wikimedia." 
93 “List of Wikipedias." 
°4 According to Alexa.com, an internet traffic ranking service owned by Amazon.com, the domain is 
HX ae: the 8th most-visited website in the United States and the 13th globally. 
< SE, 05 boyd, “Hacking the Attention Economy”; boyd, It’s Complicated. 
XX ?6 Zuboff, The Age of Surveillance Capitalism. 
?7 Papacharissi, Affective Publics. 
°8 Cohen, “Wikipedia: Exploring Fact City”; “Once Considered a Boon to Democracy, Social Media 
Have Started to Look like Its Nemesis.” 
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avoided mass circulation of unsourced or unverifiable material to the same degree 
as social media platforms such as Facebook.? The Reading Together project team is 
sympathetic to the motivations of Wikipedia editors to actively flag or cross-check 
sources in a historical moment when disinformation has been pervasive across the 
internet ecosystem.?? 


New editors are reticent to Wikipedia is distinct from other social media platforms in that 

participate using an online it relies on volunteers to govern. Moreover, Wikipedia's content 

. : has been adapted to verify information elsewhere: Google uses 

a MOIS GÉRIENE Wikimedia metadata to develop its knowledge graph. YouTube 

to a male-dominated online began linking to Wikipedia articles as an effort to counter the 

environment known for circulation of conspiracy theories. Home robots such as 

doxing and harassment. Amazon's Alexa and Google Home are built using metadata from 

Wikipedia, DBpedia (a derivative of seven language versions 

of Wikipedia), and sister project Wikidata for knowledge graphs.*? In 2020, Twitter 

announced it is using Wikipedia articles to determine which users will get a "blue 

check” verification. In 2021, Wikimedia will offer paid services to companies and 

organizations reusing Wikimedia content with Wikimedia Enterprise.** While not 

always comprehensive nor correct, the cross-validation of Wikipedia by these internet 

platforms has bolstered Wikipedia's visibility and status as an authoritative source of 
information.*° 


An individual can be a Wikipedia user in a range of ways. Anyone is free to read, use, 
and share Wikimedia content following posted Creative Commons licenses. According 
to Wikipedia's policies, anyone is free to edit and make editorial decisions. Volunteers 
who have been involved in the Wikimedia movementas editors, admins, and community 
organizers are sometimes called "Wikipedians." Wikipedians have come together to 
form movement affiliates, including "user groups and chapters," which are legalentities 
that support involvement of volunteers in the governing of Wikimedia projects. For 
instance, the non-profit ArttFeminism is among the recognized affiliate user groups. 
This constellation of editors and user groups, formal and informal, are often referred 
to as "the community.”*” However, as described in a 2020 book chapter by Sian Evans, 
Jacqueline Mabey, Michael Mandiberg, and Melissa Tamani, the "community" is not 
homogenous nor singular, but multiple and contradictory.*® 


?? McDowell and Vetter, "It Takes a Village to Combat a Fake News Army." 

10 Wardle, “The Age of Information Disorder.” 

11 Etherington, "YouTube to Add Wikipedia Background Info on Conspiracy Videos." 

1? Sengul-Jones, "The Promise of Wikidata"; Simonite, "Inside the Alexa-Friendly World of Wikidata." 

13 Harrison, "Twitter Wants to Use Wikipedia to Help Determine Who Gets a Blue Checkmark." 

1^ "Wikimedia Enterprise." 

15 Media studies scholar Jose van Dijck describes these relationships as examples of the ideological 
alignments between Wikipedia and for-profit platforms. See Dijck, The Culture of Connectivity. 

16 "Wikipedia:Wikipedians." 

17 "Sointhis sense, Wikipedians are people who form the Wikipedia Community," see 
"Wikipedia:Wikipedians." 
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Notable for this project is that Wikipedia has been criticized for systemic biases.’ 
Though "anyone" can edit, not all users who read Wikipedia contribute as editors.?? 
There are millions of page views per month, but only approximately 15,500 users 
are editors who make more than 25 edits per month in English.?* In French, there 
are approximately 2,500 users who are also editors who make more than 25 edits 
per month,?? and in Spanish only 2,200.73 Most editors self-identify as male; the 
"gender gap" present among English Wikipedia editors and content has been well- 
documented.” A 2011 Wikimedia Foundation survey found fewer than 10% of editors 
self-identified as female.?? More recent research puts that number at 16% globally and 
2396 in the United States?* Beyond editor demographics, content has been adversely 
affected. Data analysis tools and computational linguistics studies show that English 
Wikipedia has fewer and less extensive articles on women.” These same tools have 
shown gender biases in biographical articles. French and Spanish Wikipedias suffer 
from similar inequities. Other research on Wikipedia has found evidence of content 
bias that cannot be attributed to existing prejudices in secondary literature, but is the 
outcome of biases in the editing process. The 2018 Gender Equity Report identifies 
editorial rigidity as a significant barrier to achieving the Wikipedia project's vision and 
mission of providing a summation of human knowledge.?? 


"The colonial act of erasing Our effort builds upon research on Wikipedia that calls into 

cultures includes the question how participation in the self-governing community 

takes place and the effects. Some Wikipedia editors who self- 

; à identify as women exert “emotional labor" and “toll” in editing.?? 
of feeling as if you cannot . , ie "i 

gi : New editors are reticent to participate using an online interface 

and should not ‘disrupt’ the that is a gateway to a male-dominated online environment 

information architecture." known for doxing and harassment. In her 2020 book chapter, 

Alexandria Lockett. Alexandria Lockett identifies several socio-technical barriers 


psychological condition 


18 Evans et al., “What We Talk About When We Talk About Community," 224. 

1? Tn their essay, Causevic and Sengupta describe how Wikipedia, as an essential infrastructure of the 
internet, exacerbates existing inequities of marginalized communities across the world, even as the 
project, like the internet more broadly, promise to be emancipatory and democratic. Instead, the 
persistence of invisibilities on Wikipedia further entrenched marginalities through its use. Wikipedia is 
not a reflection of existing inequalities, rather, it entrenches them. See Causevic and Sengupta, 
"Whose Knowledge Is Online? Practices of Epistemic Justice for a Digital New Deal" 

Ford and Wajcman, “Anyone Can Edit’, Not Everyone Does.” 

"English Wikipedia Editors by Edit, January 2001-April 2021." 

"French Wikipedia Editors by Edit, January 2001-April 2021." 

"Spanish Wikipedia Editors by Edit, January 2001-April 2021." 

See also: "Gender Bias on Wikipedia" https://en.wikipedia.org/wiki/Gender. bias on. Wikipedia 
"2011 Editor's Survey." 

Hill and Shaw, “The Wikipedia Gender Gap Revisited"; Lam et al., "WP." 

27 Graells-Garrido, “First Women, Second Sex: Gender Bias in Wikipedia”; Wagner et al., “It's a Man's 
Wikipedia?"; Wagner et al., "Women through the Glass Ceiling." 

Graells-Garrido, et al., “First Women, Second Sex: Gender Bias in Wikipedia." 

"Gender Equity Report." 

Menking and Erickson, "The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's 
Largest Online Encyclopedia." 
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that suppress inclusive participation, specifically reflecting upon her experience guiding 
hundreds of young Black women at Spelman College, a historically Black women's 
liberal arts college in the U.S., to edit Wikipedia. First, contributing to Wikipedia requires 
numerous literacies, including the process of learning the contribution process and 
editorial pecking order.?* Lockett also names the power that the clean, authoritative 
Graphic User Interface (GUI) has over new users. There are psychological stakes of 
intervening, especially for users who fear they either do not know enough or may be 
penalized for participating in the wrong way.?? Lockett argues that the interface is a 
form of information warfare against colonized populations. "The colonial act of erasing 
cultures includes the psychological condition of feeling as if you cannot and should 
not 'disrupt' the information architecture." Wikipedia users who believe they have the 
authority to edit are those who feel welcomed and recognize themselves in the graphic 
interface and have confidence their knowledge will be recognized within this space. 


Unfortunately, the experience of recognition is an exception for most readers, rather 
than the rule, as the saying goes. As information activists and trainers, we are well 
aware of the user interface and psychological difficulties that newcomers face as 
the question of authority has not only been our own experience but is present the 
experiences of the participants in our training programs. In our respective positions 
prior to undertaking this research project, we have piloted new ways of training 
participants, including orientations to Wikipedia's culture and workarounds. This 
research project identifies the ways that organizational values and processes around 
reliability and the reliable source guidelines are implicated in maintaining hierarchies 
and excluding marginalized knowledges and communities. This effort challenges the 
mythology that Wikipedia's user-governance process is a complete success story. 
Instead, we name invisible layers of labor, the barriers, the lack of rigor, and the effects 
of the guidelines as they are currently enacted. 


«c 


31 Ford and Geiger, “Writing up Rather than Writing down’: Becoming Wikipedia Literate.” 
?? Lockett, “Why Do I Have Authority to Edit the Page? The Politics of User Agency and Participation on 
Wikipedia." 


METHODOLOGY 


Understanding the This research project was guided by an intersectional feminist 
circumstances and values epistemology and methodology. Intersectionalfeminist research 
through which legitimacy is committed to questions of power, including how differences are 
: created and reproduced. While united in struggle, scholars and 

is conferred, and at what a os | 
Í . activists before us have taken diverging analytical lenses in the 
costs, is crucial to the effort effort to understand and remedy oppressive structures of power. 
to remediate asymmetrical ^ we build on efforts in feminist technoscience to acknowledge 
structures of power. that, epistemologically, knowledge is "situated" and knowers 
have a “partial perspective.”*3 In other words, there is no such 
thing as purely "objective" knowledge because there are no pure, objective knowers. 
Rather, there are practices of knowing. We foreground asking "who is the knower and 
who is being known" in efforts to determine reliable sources.%* Understanding the 
circumstances and values through which legitimacy is conferred, and at what costs, is 

crucial to the effort to remediate asymmetrical structures of power. 


The concept of "situated knowledge" is central to our methodology. We reclaim offwiki 
processes and conversational space as legitimate and used them as opportunities 
to generate new forms of knowing about how reliability might be understood and 
alternative ways of knowing can be shared. We drew on the collective expertise of 
our own experiences, that of the members of the project's advisory committee,** and 
participants in community conversations as our research data collection. 


Who are we? 


As researchers, we are embedded in the communities we are studying. All of us are 
Wikipedia editors: Berson, Sengul-Jones, and Tamani have fluency in French, English, 
and Spanish, respectively. In addition, we are experienced as trainers and feminist 
information activists. Berson and Tamani are co-leads of Art+Feminism. Sengul-Jones 
has independently led and supported numerous training initiatives over the past decade, 
including Art+Feminism events. We used the Art+Feminism Safe/Brave Space policy to 
run our meetings.°° 


33 Haraway, "Situated Knowledges: The Science Question in Feminism and the Privilege of Partial 
Perspective"; Hill Collins, Black Feminist Thought. 

34 Hesse-Biber, Handbook of Feminist Research, 5. 

35 Biographies of the members of the AC are listed at the end of this report. 

356 “Brave Space: Remember the common goal. 2. Make no assumptions. 3. Use all-gender inclusive 

CX ? language. 4. Put accessibility at the center. 5. Honor everyone's boundaries. 6. Confront harassment 

EXE and reduce harm. 7. Make your support and safety team visible. 8. Embrace your mistakes, then 

E move forward. 9. Be brave in holding others to account, then move on." 

See https://artandfeminism.org/resources/safety/safe-space-brave-space/ 
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UNRELIABLE GUIDELINES | METHODOLOGY 


Art+Feminism is an international nonprofit organization committed to closing 
information gaps related to gender, feminism, and the arts, beginning with Wikipedia. 
Based in the United States, the co-leads of the international Art+Feminism community 
hail from Peru, French Canada, and Ghana. Since 2014, more than 1,260 Art+Feminism 
events have taken place around the world, with more than 18,000 people participating. 
Art Feminism's flagship program is a do-it-yourself campaign to create and improve 
articles on Wikipedia and its sister projects. There are 21 regional ambassadors and 
community organizers, as well as many institutionally supported librarians, curators, 
and cultural industry workers who champion the Art-- Feminism community campaign, 
along with tens of thousands of followers in social media spaces. 


Research participants in Reading Together include five members of the multilingual 
Advisory Committee and 52 participants in three community conversations (Town 
Halls) about their experiences with Wikipedia between 2015-2020. The majority 
of participants in these sessions had led at least one Wikipedia editing or training 
initiative. Participants were librarians, trainers, educators, editors, and Wikipedians- 
in-Residence. The community conversations were a time for movement leaders to 
share success stories and anecdotes, air grievances, and brainstorm what they would 
like to see to enact knowledge equity on Wikipedia. 


What we did 


Reading Together had three parts: three Town Hall community conversations, close 
readings, and this report, which includes recommendations for a more equitable 
Wikipedia. Throughout the process, we have solicited written and oral feedback from 
the Advisory Committee. 


Close reading of 
Reliable Sources 


Community Conversations 


Guidelines 
: : - Town Halls in English, French and Spanish - 
in English, French and 
Spanish Report 
Interpretation 
and 


Recommendations 


Components of the research project Reading Together: Reliability and Multilingual Global Communities 
(September 2021 - March 2021). 
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Who is this report for? 


As an Art+Feminism project funded by WikiCred, this report is for the multilingual 
Wikimedia community interested in critical approaches to reliability from the 
perspective of both credibility studies and antiracist and intersectional feminism. Our 
recommendations are specific to Wikipedia's current community processes, however 
we anticipate current and new editors, trainers, librarians, journalists, artists, activists, 
and academics in the social sciences and humanities will find this report of interest. 
More broadly, this report will be of interest to people and organizations concerned with 
an equitable, multilingual internet society. 


Close reading 


We conducted close reading of relevant community pages. We also analyze case 
studies brought to us in the community conversations. 


Community conversations 


AS In November 2020, we advertised and ran three open Town Hall community 
E. conversations in French, Spanish and English. They were between one and a half 
hours and three hours. We had 52 participants total. Conversations were administered 
in three languages, and translation services were provided by Cenzontle Language 
Justice Cooperative. The conversations were recorded and transcribed for the purpose 
of this research. We also gathered feedback via email and online survey responses. 
Excerpts from the community conversations are included in this report. Participants 
have been anonymized. 
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OVERVIEW OF GUIDELINES 


Community pages about reliable sources in French, and Spanish were translated from 
the English version, which was created in 2005. They have diverged in content since 
however similarities persist. All three define a reliable source as a publication that is 
independent from the subject and suggests privileging of academic presses. Citations 
are not used in the guidelines or essays, as community consensus is presumed. The 
next sections summarize each language version in greater detail. 


English Wikipedia 


Summary - Reliable Sources Guidelines in English Wikipedia - 


Official guideline Content Guideline 


Status Project Page "Reliable Sources" - Shortcut: WP:RS 
https:;//en.wikipedia.org/wiki/Wikipedia:Reliable sources 


Date of creation 2005 
Largest number of edits between 2006 - 2009 
Greatest editorial changes between 2011-2020 


Definition of No 
reliability 


Complementary "Reliable sources Noticeboard" 
pages https://en.wikipedia.org/wiki/Wikipedia:Reliable sources/Noticeboard 


“List of Perennial Sources" 
https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources 


The English Reliable Source guideline article page begins by stating that Wikipedia 
articles should be based on verifiable sources that are reliable, independent, published, 
and have a reputation for fact-checking and accuracy. A source may be a piece of work 
(book, article), the creator of the work (author), or the publisher of the work (a press). 
Reliability and unreliability are not defined in the guideline. The definition of published 
is broad: “materials that have been recorded then broadcast, distributed, or archived 
by a reputable party may also meet the necessary criteria to be considered a reliable 
source.” The criteria mentioned, however, is not specified. Editors are cautioned 
against using unpublished or self-published sources, as well as commercial sources, 
predatory journals, and press releases. The guideline offers advice on how to interpret 


sources (e.g. four types of sources, questionable and self-published sources, and 
reliability in context) to make decisions about reliability. 


The Reliable Source guideline article page was started on February 28, 2005. The 
information in the guideline has changed overtime, but the page has retained the same 
structure over the past decade. The largest number of edits were made to the page 
from 2006 through 2009. The greatest editorial changes, indicated by growth in size by 
bytes from 20,000 to 50,000, took place from 2011 through 2020.3” New contributions 
to the page in the past decade have been reverted due because editors ask for new 
changes to community consensus. These contributions have included suggestions to 
. — expand the guideline to include self-published materials and 
Ultimately, the guideline experts weighing in on discussion boards or social media.?* 
foregrounds the role of 

interpretation by Wikipedia Only about fifteen sources are used to develop this guidance 
editors in deciding what ©" evaluating the reliability of sources. There is no definition 
. of reliability or a literature review on the history or cultural 

counts as a reliable source. ome | ie 
an . specificity of the concept. According to the English Wikipedia 
Editors must use their community, guidelines do not require references. Policies 
judgment to draw the and guidelines are not a part of the encyclopedia. These 
line between usable and texts are developed through collaboration and governed by 
unreliable sources.” community consensus.?? Consensus is defined as a “normal 
and invisible process" that "naturally" happens between 
editors.^ Editors participate until they reach a resting point, after which, silence is 
presumed to mean consensus.^' We'll analyze the significance of this process for new 

editors and marginalized communities in the findings. 


Ultimately, the guideline foregrounds the role of interpretation by Wikipedia editors in 
deciding what counts as areliable source. "Editors must use their judgment to draw the 
line between usable and unreliable sources." The page, which is approximately 5,000 
words, also is a landing page for more than fifty wikilinks that editors are suggested to 
consult, including the Wikipedia policy on Verifiability, a Reliable Source noticeboard, 
additional essays, and the Perennial Sources list. 


w 
SI 


See Figure 1.1. 

"Talk: Reliable Source Guideline." 

"The policies, guidelines, and process pages themselves are not part of the encyclopedia proper. 
Consequently, they do not generally need to conform to the same content standards or style 
conventions as articles. It is therefore not necessary to provide reliable sources to verify Wikipedia's 
administrative pages, or to phrase Wikipedia procedures or principles in a neutral manner, or to cite 
an outside authority in determining Wikipedia's editorial practices. Instead, the content of these 
pages is controlled by community-wide consensus, and the style should emphasize clarity, 
directness, and usefulness to other editors." See "Wikipedia:Policies and Guidelines." 

"Silence and Consensus." 

41 “Silence and Consensus." 
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French Wikipedia 


Summary - Reliable Sources Guidelines in French Wikipedia - 


Official guideline There isn't a specific guideline about reliable sources. The closest 
recommendation is the page "Citez vos sources" (in English, "Cite your 
Sources"). - Shortcut: WP:CVS 
The page was started as a translation of the 2004 version of the English 
Wikipedia content guideline "Citing Sources". 
https:;//fr.wikipedia.org/wiki/Wikip96C396A9dia:Citez vos sources 


Status Recommendation 


Date of creation 2004 


Definition of 
reliability 


Complementary Essay "Sources Fiables" (in English, “Reliable Sources"). 


pages The essay was started as a translation from the 2011 version of the English 
Wikipedia content guideline “Reliable Sources”. 
hitps://fr.wikipedia.org/wiki/Wikip%C3%A 9dia:Sources_fiables 


"l'Observatoire Des Sources" (in English, “Observatory of Sources"). 
https://fr.wikipedia.org/wiki/Wikip96C396A9dia:Observatoire des sources 


French Wikipedia is distinct from English Wikipedia and Spanish Wikipedia 
because there is no specific guideline about reliable sources. The closest guideline 
(recommendation) in French Wikipedia is the Citez Vos Sources page (Cite Your 
Sources), which was created in 2004asatranslationfrom the English project page Citing 
Sources. This is the page where new users are directed when looking for information 
on creating articles. Like other guidelines, it requires community consensus.? The 
page Is written as a series of answers to the question of "Why cite sources?" The short 
answers provide editors with general guidance on citations and their importance for 
an encyclopedia. There is a brief bibliography, which is self-referential in nature: the 
publications cited are about Wikipedia's citation practices. There is a short section 
on quality and problematic sources, including a statement that blogs are not reliable. 
Finally, and more recently, the Citez Vos Sources subsection on evaluating the quality 
of sources is a redirect to an essay, Sources Fiables.^ 


^? A "recommendation" is the French equivalent of a guideline and uses community consensus to 
achieve authority. 

43 Whereas Citez Vos Sources has been relatively consistently but quietly edited by a small but 
dedicated group of users since its creation, Sources Fiables has experienced low numbers of edits 
since 2013, until 2020 when it was heavily edited by user Nouill. Sources Fiables contains a redirect 
back to Citez Vos Sources, added by user Nouill on April 14th, 2020. 


This essay, Sources Fiables, is a translation of the English Reliable Sources guidelines 
article page, which was translated from English in 2011.^ “Tn French Wikipedia, essays 
are not supported as policies or guidelines and have no official status in the Wikimedia 
community. While an essay may "addresses a particular aspect of the functioning 
of Wikipedia, [...] it does not necessarily represent the opinion of the community, 
but primarily the opinion of its authors.” Sources Fiables is approximately 4,000 
words and includes detailed information about the importance of sourcing for the 
encyclopedia and suggestions on ways to evaluate source reliability, including context 
and cross-checking. Editors are cautioned to not use sources that may have been 
sponsored or that lack editorial oversight, including conference inserts and academic 
journal supplements. While the essay receives fewer page views than the Citez Vos 
Sources, there is crossover material. In April 2020, a survey was conducted on French 
Wikipedia to determine whether Sources Fiables should become a recommendation. 
The results were closed as inconclusive.** 


Consensus is defined as As it stands, the official guideline Citez Vos Sources and the 

a “normal and invisible essay, along withanoticeboard andathird site Observatoire 
Des Sources, are among the pages that editors may go for 
guidance on evaluating reliable sources. Yet, across these 
sites of information, there are inconsistencies. For example 


process" that "naturally" 
happens between editors. 


Editors participate until they Citez Vos Sources states that blogs are not reliable sources 
reach a resting point, after and should not be used on French Wikipedia. Meanwhile 
which, silence is presumed the Observatoire Des Sources condones some blogs, but 

to mean consensus. cautions against allotting disproportionate importance to 


these sources (Wikipédia:Importance disproportionnée). 
The project page on Sources and Chez Manon include discussions on blog reliability 
under a section on self-published media but the conversations are inconclusive. “Blogs 
can be acceptable sources if their authors are journalists or professionals in the field 
on which they write [...] self-published expert sources can be considered reliable when 
they are produced by an established expert on the subject of the article and have been 
previously published by serious third-party publications." ^? 


Since their creation, the official French guideline Citez Vos Sources has received the 
highest number of page views, with a daily average of approximately 1,200. The page 


I 
R 


"Sources Fiables." 

^5 As described in the previous section, on English Wikipedia, the Reliable Sources guideline is a content 
guideline that supports the three core content policies. A policy has more weight than a guideline. 

46 See "Project namespace" and "Essai." Essays in Spanish and English do not require community 
consensus either. 

(Translation author's own) "Essai." 

(Translation authors own) "Sources Fiables." "The survey, which was open from March 31 to April 30, 
2020, did not yield any results. A majority of contributors were in favor of maintaining the trial status. 
Several Wikipedians have mentioned that the page has yet to mature, others are in favor of maintaining 
the test in the long term because several pages of recommendations already address the subject." 
"Sources Fiables." 
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has had edits from 167 distinct editors since recording began in 2015. However, 
it is important to highlight that Citez Vos Sources contains information on citations 
that is outside the scope of reliability, it is not possible to accurately determine what 
percentage of page views of or edits to Citez Vos Sources pertain to the section on 
reliability. Meanwhile, the essay Sources Fiables has received far fewer average page 
views, the daily average is five, and there have been 28 editors since the page was 
created.50 5: 


Spanish Wikipedia 


Summary - Reliable Sources Guidelines in Spaish Wikipedia - 


Official guideline Page "Fuentes fiables" (in English, "Reliable Sources"). 
Shortcut: WP:FF 
The page was started as a translation of the 2008 version of the English 
Wikipedia content guideline "Reliable Sources". 
https://es.wikipedia.org/wiki/Wikipedia:Fuentes fiables 
https://fr.wikipedia.org/wiki/Wikip 96C 396A 9dia:Citez_ 


Date of creation 


Definition of 
reliability 


Complementary There are none 
pages 


On Spanish Wikipedia the Project Page Reliable Sources (in Spanish, Fuentes fiables) 
was initiated in October 2008, as a translation from the English Wikipedia project page 
Reliable Sources.*? In December 2009, the page was stated as an Official Policy, which 
means that it's content is mandatory for all Wikipedians and, like English Wikipedia 
above, any attempt to make significant changes should have prior community 
consensus. The hierarchy of editorial rules in this language version is headed by 
"policies," followed by "guidelines or conventions," which describe recommended 
good practices. Along with Wikipedia is not a primary source,* Verifiability,^ Neutral 


50 “Pageviews Analysis: Wikipedia: Sources Fiables and Wikipedia: Citez Vos Sources." 

51 See Figure 1.2. 

? Spanish Wikipedia. “Fuentes fiables." 

? Spanish Wikipedia. "Wikipedia no es una fuente primaria." 

^ Spanish Wikipedia. “Verificabilidad” Spanish Wikipedia. “Wikipedia no es una fuente primaria." 
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Point of View,” and Biographies of Living People,” these four policies constitute the 
main editorial policies regarding the types, quality and forms of use of the sources 
within Spanish Wikipedia. 


In the Spanish policy, reliable sources are defined as "works by authors who are 
accredited in the field in question." The authority of a source is determined by the 
publication process, which certifies the veracity of the information based on the 
degree of acceptance and use of the source by third parties. To be considered reliable, 
sources must be "independent" from their subject and published. The absence of 
reliable sources on a topic may be a reason for deciding not to include it in Wikipedia 
as an article. 


The policy addresses the degree of reliability of specific varieties of sources: primary, 
secondary and tertiary; specialized publications; press and self-publications. It 
indicates the degree of acceptability of these types of sources by Wikipedia and the 
specific ways in which they can be used. For example, primary sources and self- 
publications aren't considered reliable in general, but they can be used rarely to 
provide information of general nature or about themselves. Some self-publications can 
be considered reliable if they are produced by persons or entities "of widely recognized 
authority on the subject" or if their content is based on reputable sources. It is not 
specified whether this accreditation should match the rules established in the policy 
itself or the credibility criterias used in other fields (e.g., academia or the media), thus 
creating room for interpretation by the reader. 


Itis relevant to observe how one of the rules was defined taking into consideration that 
university and peer-reviewed publications are not always available to Wikipedians, 
so the use of sources with less strict verification standards is allowed. This includes 
what is named as “the rest of the specialized publications," especially if they have “a 
consolidated reputation in their field," and the monographs made by "authors with an 
accredited curriculum in the respective field." 


The policy is not presented as a set of rigid rules, there is considerable room for 
interpretation and judgement of the reliability of the sources by the reader. The only 
case in which categorical terms are used to forbid the use of certain types of sources 
is in reference to Biographies of Living People. In the other cases, the policy takes on 
a recommendatory or character, which is expressed by the recurrent use of language 
constructions such as "attempts will be made to use reliable sources;" "special 
care should be taken;" "the absence of sources may be grounds for deletion of an 
article." In some cases the term "must" is used (e.g. "Sources must directly support 
the information," "Wikipedia articles must use reliable secondary sources"), however, 
exceptions to the above rule are usually mentioned afterwards. 


55 Spanish Wikipedia. “Punto de vista neutral.” 
56 Spanish Wikipedia. “Biografías de personas vivas." 
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The View History tab” and the Tool History Statistics®® provide information about the 
collaborative editorial process, the maintenance and volume of readings of the policy. 
The editorial process consisted of the direct translation of content, discussions around 
the interpretation of specific terms, addition of original content, the definition of the 
page structure and the rigidity of its enforcement. 


Until November 2020, the total number of edits on the page since its creation was 410. 
The total number of editors was 54, not counting editions from IP addresses and bots. 
Minor edits makeup 68.596 of the total (281). Most of them were made in 2009 (250). 
The frequency of editing is considerably reduced thereafter, fluctuating between 20 
and 1, which is the case for 2013, 2015 and 2018. In other words, the main content 
of the policy has remained unchanged for 12 years.* As for the number of visits, using 
as a sample the range between October 27 and November 26 2020, the daily average 
is 590 and the total number is 18,271.° Taking into account the same range of time, 
visits to the Reliable Sources policy exceed by far visits to related policies.ft 


a 
ES] 


Spanish Wikipedia. "Historial de «Wikipedia:Fuentes fiables»." 

58 Wikipedia User Aka. “History statistics for ‘Wikipedia:Fuentes fiables’.” 

5° See Figure 1.3. 

6° Wikipedia Users Musik Animal, Kaldari, Marcel Ruiz Forns. “PageViews Analysis, Wikipedia:Fuentes 
fiables.” 

61 Between October 27 and November 26 2020, the “Neutral Point of View” Policy had 1989 visits, and 

“Wikipedia is not a primary source” had 8,655 visits. 
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FINDINGS 


These findings read together the English, French, and Spanish Wikipedia's 
conceptualizations of reliability as located in the pages themselves and found in the 
SOS lived experiences and perspectives of editors, trainers, and educators. The evidence 
dC Lea we provide of the features of the reliable source guidelines are not found in individual 
c page histories or guideline language alone but are located at the intersection of the user 
experiences with these guidelines, the reports of labor and interpretation by trainers 
and educators to make the many guidelines of Wikipedia legible to newcomers, and 

selected case studies of articles that were called into question or challenged. 


The guidelines do not I, Lack of rigor 


cover the history ofthe — The guidelines about reliable sources in English, French, and 

concept of reliability ^ Spanish lack academic rigor, notably through the lack of 
or the epistemological citations to support the claims. The English, Spanish, and French 
guidelines, and the French essay, emphasize that academic 
sources are a gold-standard of reliability, though such sources 
are not used in the construction of the guidelines. The Spanish 
guidelines notes that university and peer-reviewed publications 
are not always available to Wikipedians, which is why the use 
of sources with less strict verification standards is allowed, such as "the rest of the 
specialized publications," especially if they have "a consolidated reputation in their 
field," and the monographs made by "authors with an accredited curriculum in the 
respective field." 


criticisms within the 
French, English, and 
Spanish academies. 


Thus, the documentation in the guidelines provides for ways that Wikipedia editors 
might consider reliability in their efforts to edit, but the same standards are not 
applied to the guidelines themselves. The guidelines do not cover the history of the 
concept of reliability or the epistemological criticisms within the French, English, and 
Spanish academies. These are schools of thought that call into question the ways 
that mainstream academic presses rely on Enlightenment concepts (or in French, the 
ideas of liberté, égalité, fraternité). Historically, these have excluded perspectives and 
reinforced historical inequality.*? Prestigious institutions and custodians of research 
and knowledge—including academic libraries, textbook publishers, categorization 
and indexing standards—have been shaped historically by legacies of injustice 
and erasure.9 Their authority is contextual and constructed. Consequent biases in 


62 Henry, Tator (eds.) Racism in the Canadian University: Demanding Social Justice, Inclusion, and 
Equity, and Cesaire, Discourse on Colonialism. 
65 Christen, “Tribal Archives, Traditional Knowledge, and Local Contexts: Why the ‘S’ Matters." 
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> secondary literature can take the form of misrepresentation or absence, adversely 
E affecting marginalized communities due to the limits of existing sources, and access to 
M them. These limitations go unmentioned in all three language versions of the guideline, 


In all three languages, editors are cautioned to avoid commercially-motivated 
206 publications, including sponsored content, supplemental materials, press releases, 
TEN "churnalism" and human-interest features. While scholarship, especially textbooks, 
% and news media are acceptable.‘ This definition leaves out the ways that some 
commercially-motivated materials can include factual information, while scholarship 
and news media can reproduce biases and marginalizations through claims to 
objectivity. As one participant identified in a Town Hall conversation: 


This definition can obscure the political and economical agendas behind 
scholarship and academic research that influence research lines, topics, 
budgets, dissemination and reputation, impacting knowledge availability 
around marginalized subjects. 


Participants in all three community conversations described this lack of rigor and the 
subsequent biases as problematic. One participant said in a recorded conversation 
that the supposed universality of these suggested sources is frustrating for activists 
concerned about indigenous communities, who have been misrepresented. In general, 
they went on: 


There is no mention of problems of systemic biases in secondary sources in 
either English, French or Spanish. The problem of the existence of systemic 
biases in the construction of sources, especially secondary sources, is not 
mentioned in the content of the policies, nor in their discussion page. 


A trainer in the conversation on Spanish Wikipedia found the lack of attention to 
criticisms about reliability missing from the guidelines especially troublesome because 
it meant Wikipedians didn't have a frame of reference for epistemic criticisms: 


There is no section on non-hegemonic communities or issues with little 
representation, [no] guide for Wikipedians on how to address these issues.°° 


64 “Human-interest reporting is generally not as reliable as news reporting.” There is a citation for this 
claim, but the citation undermines the claim. The citation is to a book review by Laura Miller, where 
in the review, Miller notes that the author of the book under consideration includes a section where 
that author reported that one of her interviewees said that they exaggerated (“took considerable 
license") in their human-interest reporting. This caught our attention as an unconvincing, unreliable 
source to support a general claim about the said rigor or reliability of human-interest articles. 
Inquiries about the lack of rigor of this source on the Talk page resulted in a reminder that policies 
and guidelines are based on consensus, not sources. 

See the next section for further discussion of consensus. 

55 "No existe una sección sobre comunidades no hegemónicas o temas con escasa representatividad, 

una guía para wikipedistas acerca de cómo abordar estos temas." 
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As one participant in the session concerning French Wikipedia said, this emphasis 
privileges French perspectives and consequently marginalizes Quebec-based users. 


It's so focused on the Académie Francaise, arts, letters, etc., that there is a 
French system that doesn't represent itself for Quebec artists, so its really 
hard for our pages to be accepted or to have credibility.°° 


There was agreement among participants in the Town Hall about French Wikipedia 
in French that "the information is very clear. It is when you get into specific details, 
specific disciplines, that it becomes more complicated." Another participant noted 
that the reliability "regulations need to be adapted for different disciplines." For arts 
and culture disciplines, areas of expertise not mentioned in the guideline, trainers 
experienced having their, and their trainees, expertise questioned by editors who were 
interpreting a guideline rather than a corpus of knowledge. 


[The guidelines and essay] are very clear but not applicable to all 
environments. We are used to creating pages in the arts community. People 
who recommend these pages for removal are not people who know reliable 
sources in the arts community. An example catalog is a reliable source. 
Some media. It's as if the guidelines are too general and don't apply to the 
nuances of different media.°’ 


A trainer who organizes editathons at a feminist cultural center for Spanish Wikipedia 
similarily reported on the problems her participants encounter with sourcing and a lack 
of understanding among editors about the sources they were using for their articles: 


When they work on articles about women they run into the problem of 
reliable sources in various Wikipedias. With biographies, but also about 
topics like motherhood, breastfeeding.9? 


Others pointed out that sources that are not necessarily unreliable are cast as such 
according to the guidelines: 


66 "C'est tellement axé sur l'Académie française, arts, lettres etc., qu'il y a un système francais qui 
ne se représente pas pour les artistes de Québec, donc c'est vraiment dur pour que nos pages soient 
acceptées ou qu'on ait de crédibilité." 
57 “[Ces lignes directrices et l'essai] sont très clairs, mais ne s'appliquent pas à tous les milieux. Nous 
sommes habituées à créer des pages dans le milieu des arts. Les gens qui recommandent ces pages 
pour la suppression ne sont pas des gens qui connaissent les sources fiables dans le milieu des arts. 
Un catalogue par exemple est une source fiable. Certains médias. C'est comme si les lignes directrices 
sont trop générales et ne s'appliquent pas aux nuances de différents milieux." 
"Cuando trabajan artículos sobre mujeres se topan con el problema de las fuentes fiables en diversas 
Wikipedias. No solo escribe sobre biografías, sino también sobre temas como maternidad, lactancia." 


68 


UNRELIABLE GUIDELINES | FINDINGS | 24 


I would like to see a note on judgment with using blogs and online 
magazines, to weigh in favor of newer and more diverse publications that 
might not have the bona fides of traditional media. Because publishing 
has changed and traditional media sources that cover the knowledge of 
marginalized groups are less likely to emerge, blog sources in marginalized 
or "niche" topics should be used more widely. 


Trainers expressed frustration that institutionally-produced pamphlets, catalogs and 
essays are challenged as "unreliable" due to a perceived lack of independence from 
the subject, or being "promotional": 


Pamphlets [have] important information. And a curatorial essay has cited 
information. Pamphlets [can be] produced [by] institutions that have 
smaller budgets, which might actually be more likely to represent artists 
from, for example, BIPOC community. It is these spaces that actually give 
us sources on the work of these artists that are not represented in large 
museums or that won't have the traditional space in the media. 


Participants, especially librarians, also noted the lack of guidance for editors on how 
to evaluate the range of sources of secondary information such as catalogs, databases, 
indexes, bibliographies, pamphlets, online magazines or press releases, which may be 
from institutions but are not primary sources, nor are unreliable simply by nature of 
being produced by an institution. 


In summary, the lack of rigor of the guidelines results in "reliable source" and "individual 
interpretation" being treated as ahistorical and uncontroversial concepts, which they 
are not, and the guidelines reinforce their ideological power. 
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II. Silence as consensus is problematic 


“If you disagree, the onus is on you to say so." 9? 


Thelackof rigor described above is an outcome ofthe reliance on community consensus 

to write editorial guidelines. As explained in the previous section, community consensus 

controls the content of policies and guidelines for all three Wikipedia editorial 

communities. Consensus is a "normal and invisible process" that "naturally" happens 

between editors."? Editors participate until they reach a resting point, after which, 

silence is presumed to mean consensus.”* In other words, newcomers or editors who 

may feel uncomfortable by the interface are presumed to be in 

Others noted a disconnect agreement, even if they do not agree, simply because they have 

between the Wikimedia ot spoken up by pressing edit in the editorial backchannels. 

In English and Spanish, the guidelines were solidified through 

the consensus of a small number of editors. While in French, 

the page about reliable sources that is consulted frequently is 

actions of the community — an essay, an article type which does not require consensus, but 
in terms of values represents the view of a single Wikipedian.72 


Foundation's claim to support 
knowledge equity and the 


During our community conversations, participants expressed frustration at the 
exclusionary features of this foundational principle on how to govern Wikipedia, 
even those who had successfully edited in Wikipedia and made new pages or other 
contributions that were not contested. 


Silence equals consent, I think, [...] just sounds so problematic. I said 
consent, and I meant consensus' There's consent, [which means] you're 
like, well, T'm not saying anything so obviously I agree with it.’ But so many 
people don't even know that these changes are ongoing. If you're not aware 
that it's happening, then how can you talk about it? And then getting into 
talking about it on those pages can be a very toxic environment. [...] I don't 
think that silence should equal consensus. 


Others noted a disconnect between the Wikimedia Foundation's claim to support 
knowledge equity and the actions of the community in terms of values, which may 
not enact knowledge equity in the patrolling or reviewing processes, or in Articles for 
Creation or Articles for Deletion noticeboards. Said one trainer: 


o 
No] 


“Silence and Consensus.” 

“Silence and Consensus.” 

“Silence and Consensus.” 

In French Wikipedia, the English Wikipedia Reliable Sources guideline was translated from English 
to French in 2011 and linked as Wikipédia:Sources fiables. This is not a Project Page, as in English, 
but an essay. On English Wikipedia Reliable Sources is specifically a content guideline developed 
through community consensus that supports the three core content policies of Verifiability, Neutral 
Point of View, and No Original Research. See "Wikipédia:Sources Fiables (Essai)." 


Mo ol 
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I would like to see a note on judgment with using blogs and online 
magazines, to weigh in favor of newer and more diverse publications that 
might not have the bona fides of traditional media. Because publishing 
has changed and traditional media sources that cover the knowledge of 
marginalized groups are less likely to emerge, blog sources in marginalized 
or "niche" topics should be used more widely. 


Anew editor wrote in an email that they perceived Wikipedia's reliable source guidelines 
to have exclusionary features.7? 


Honestly this issue is one of those innocuous institutional barriers that seem 
harmless but actually shuts many, many people out. Because oftentimes, 
they operate in tandem with other barriers and end up creating an echo 
chamber of exclusion. [...] So whether Wikipedia wants to acknowledge 
this or not, this policy contributes to a long-standing process of historical 
erasure under the guise of academic integrity and rigor. 


On French Wikipedia, the editorial process has an added layer of exclusion due to the 
treatment of the essay, Sources Fiables, which is longer and more developed than 
Citez Vos Sources, as a de facto extension of the Citez Vos Sources guideline. A small 
number of editors have participated in the community consensus process to develop 
Citez Vos Sources, while one editor has inserted a section taken, and linked to, the 
essay, which does not use consensus. 


In summary, community consensus is afoundational pillar of the Wikimedia movement. 
We learned trainers see this process as privileging those who participated first in the 
development of the encyclopedia's editorial back channels. As well, the participants 
in our community conversations were uncomfortable with the presumption that 
agreement is communicated through silence, which privileges those who have the 
time and feel comfortable speaking up and participating in editorial conversations. 


III. The essential work of trainers 


The user interface of Wikipedia (across all three languages) is a known barrier to entry 
for new editors. From the abundance of information to navigate to using Media Wiki74 
to contribute to community pages, new users have many cultural-technical norms to 
learn to not only contribute to articles, but to participate in Wikipedia's editorial 


73 Note the editor uses the word “policy” instead of “guideline,” as they were unfamiliar with the 
hierarchy of editorial rules. 
74 Media Wiki is also known as Wiki markup or Wiki code. 


backchannels.”* Though guidelines and policies are publicly visible, they are difficult to 
locate for newcomers, which can exacerbate inequalities in participation.”° 


pce à By and large our To expand the editor pool and diversify the content on 
FED community conversations Wikipedia, more experienced Wikipedians have become 
trainers to welcome and guide newcomers to navigate 
editing. There are many trainers and training programs. For 
instance, Art+Feminism has devoted considerable resources 
to developing guides and providing a welcoming community 
to orient newcomers. More generally, Wikimedia Chapters 
include information ^ and User Groups, universities, libraries, museums, and 
into Wikipedia. galleries may have staff or volunteers who work as Wikipedia 
trainers; some have more official statuses as Wikipedian-in- 
Residence. Trainers help newcomers get started and provide guidance, resources, 
and technical and support with editing. This essential labor provides newcomers with 
access to Wikipedia editors who are eager to champion and support their good-faith 
contributions. 


showcased the work 
that trainers do to 

help participants and 
institutions successfully 


The majority of our participants in the community conversations were trainers in 
some capacity: Wikipedians-in-Residence, librarians, curators, Wikimedia chapter 
or user group representatives, or leaders or support staff organizers who have led 
Wikipedia community training and public editing events. The centrality of trainers to the 
success of new users was emphasized to us in our research period by a reversal--the 
nonpresence of trainers and the reported experience of a new editor. A faculty member 
at a university in Canada received the Art« Feminism newsletter and reached out to 
share an experience. A contributor to both English Wikipedia and French Wikipedia, 
they had used the Articles for Creation tool to "submit" an article with 25 sources for 
review in English Wikipedia.” A reviewer replied by suggesting that the article should 
not be published, but needed more reliable sources to sufficiently meet the notability 
guideline.7? A second commented the article likely met the General Notability Guideline, 
though they did not move the article to mainspace. When we received the email from 
the original editor, we saw evidence of a common experience for new editors: the new 
editor appeared to see the suggestion as judgement; in addition, they disclosed later, 


75 Wagner et al. It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia, 461. 
See "Wikipedia:Gender Bias," Pandi "Shedding Light on Women Who Edit Wikipedia," and Kleefield 
"Wikipedia's Gender Bias — and What Your Students Can Do About It.” 
"Wikipedia:Pamela Edmonds;" "The Articles for Creation (AfC) process is designed to assist any 
editor create a new page as a draft article, which they can work on and submit for review and 
feedback when ready. The AfC process must be used by non-registered users and by those who do 
not yet have sufficient editing experience because these groups of editors are not permitted to create 
articles directly in Wikipedia's mainspace. The AfC process should also be used by anyone with a 
conflict of interest." From Wikipedia: Articles for Creation. 
SEX EX The General Notability Guideline is a “test” on English Wikipedia that editors use to determine if a 
E XX topic merits an article. The "test" requires a subject is covered in multiple, independent reliable 
POOR secondary sources over a considerable amount of time. 
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they did not know how to technically move the article to main space. When we learned 
about this, one of us moved the article for the editor. 


This example highlights the importance of hands-on trainers, who serve as mentors 
to newcomers, in scaffolding the editing experience. By and large, our community 
conversations showcased the work that trainers do to help participants and institutions 
successfully include information into Wikipedia. Trainers relayed how they helped 
newcomers learn the culture of Wikipedia, specifically the way that guidelines around 
sourcing are monitored and how to respond if their article is flagged for deletion 
by another user. On all three encyclopedias, new editors have the fewest privileges 
and may face scrutiny from editors with special editorial privileges, such as Pending 
Changes Reviewer or New Page Patroller. Users with special access to tools provide 
an additional layer of judgement. Gaining special user access depends on the level: 
Admins are selected through a lengthy peer-review and question and answer. New 
Page Patrollers can self-nominate and to be selected, are expected to have had an 
account for 90 days and have made 500 uncontested mainspace edits. 


Meanwhile, trainers are often hired or selected through either self-nomination, or 
brought on by a partner organization. Some trainers also have special access to tools 
themselves. Other trainers may have general knowledge about the layers of editorial 
governance, such as how articles are challenged through the Articles for Deletion 
noticeboard, or how to move a draft to mainspace, and have not or do not wish to 
seek additional special user access tools due to concerns about time and hostility. 
Nevertheless, despite their alignment in the effort to improve the encyclopedia, 
friction can arise between trainers and users who patrol new contributions. As two of 
our participants in French Wikipedia explained: 


Every time we do a Wiki day, it is always work for us afterwards, fighting for 
the pages that are deleted."? 


For scientists [biographies] I have very few problems with pages to delete. 
But in our workshops, when we work on artist pages, yes, it's much more 
complicated and makes us delete a lot of pages.®° 


In English, contributors from English-speaking countries in Africa said their contributions 
often faced scrutiny. One organizer from an unnamed African country who participated 
in our session said when they hosted events, contributions were deleted en-mass for 
lacking reliability. This was demoralizing for the participants and required extra work 
by the trainers to stand up for their publications and citations, said one participant. 


79 “A chaque fois qu'on fait des journées Wiki, c'est toujours un travail supplémentaire pour lutter pour 
les pages qui sont supprimées par la suite." 

8° “Pour les [biographies de] scientifiques, j'ai trés peu de problémes avec des pages à supprimer. Mais 
dans nos ateliers, quand on travaille sur des pages d'artistes là, oui c'est beaucoup plus compliqué et 
on se fait beaucoup supprimer de pages." 
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Ithink some of the English admins are not being fair to some of us in Africa, 
what we consider reliable [is] not reliable to them. It's very difficult to 
understand why they think so. 


To avoid new editors experiencing these disappointing responses, other trainers 
in English Wikipedia explained they would review sources before new editors begin 
editing. This could lead to bad feelings, so they explained how they were careful to 
keep up morale. 


Well, you might have to tell them they have to remove half the information 
because they don't have a reliable source. They can take it really poorly, 
shut down, not want to edit Wikipedia anymore--like this defensive, 
conflictual stance. So I've started approaching it as, okay, so you want to 
edit Wikipedia and it's really exciting. Here is a list of like really absurd 
rules you know have to look at. I kind of almost, you know, depending on 
the person or the group or whatever, approach it like it's kind of a game. 


To become a New Page patroller, users nominate themselves. To be selected, users 
"are expected to have been registered for at least 90 days and to have made at least 
500 uncontested edits to mainspace articles." Art-- Feminism has previously petitioned 
to change the retained active editor guidelines, as they previously resembled "a re- 
inscription of a traditional hierarchy of gendered labor [...] The effect is that the current 
guidelines systematically exclude new users, including people from marginalized 
communities new to the Wikimedia userspace.’® As a result of this petitioning, 
Wikipedia's guidelines for active editors were changed. On French Wikipedia there is 
an extra layer of difficulty as the regulations and those who patrol them replicate a 
colonial standpoint on French culture. As one French conversation participant stated: 


There is a perceived tension between Wikimedia France and Wikimedia 
Canada (run by French Canada); French speaking Africa; and French users 
in the U.S. 


One participant based in North America said they only create new pages at night when 
new page patrollers in France and Switzerland are likely to be asleep. 


Further, the quantity of material that a trainer is required to parse in relation to reliable 
source guidelines is immense. As one participant said: 


81 Evans, Sian, Jacqueline Mabey, Michael Mandiberg, and Melissa Tamani. “What We Talk About When 
We Talk About Community." 

82 “Il y a une tension perçue entre Wikimédia France et Wikimédia Canada (gérée par le Canada 
francais); l'Afrique francophone ; et les utilisateurs francais aux Etats-Unis.” 
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This bushy structure makes the guidelines pages unreadable. Who has the 
time and the meticulousness to read it completely without being lost to a 
certain point? It took me a decade to go through it [in French] and I must 
admit I'm not done yet! 


In summary, our community conversations emphasized the essential work of trainers 
whoare actively supporting new editors to make contributions. Trainers are themselves 
essential technologies who take on the labor of learning the guidelines, policies, and 
technical processes, which are numerous, to support newcomers. In addition, they 
can diffuse friction between on-wiki editors who are patrolling pages and newcomers 
who may be less familiar with the editorial processes. 
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RECOMMENDATIONS 


The researchers recognize the complexity of enacting change on these large, 
distributed, user-run platforms. While there is a desire in the movement for things 
to change in the spirit of knowledge equity, the reality is that change is difficult. In 
part because many of the infrastructural norms in place that we've identified. We have 
shown that the current guidelines and policies lack rigor, which allows for editors to 
prefer certain kinds of sources, which may uphold Western knowledge formations. We 
have shown that the consensus process is exclusive by presuming all editors embody a 
subject-position where it is easy—technically and psychologically—to intervene. There 
are high psychological stakes to intervening, this definition of consensus enshrines the 
guidelines as they are crafted. Finally, we have shown the crucial work that trainers 
do to scaffold the experience of new editors, forming an invisible and unrecognized 
layer of editorial work. The movement, via the 2030 Movement Strategy, is already 
engaged with processes of moving towards greater knowledge equity.?* We have 
recommendations for this community in the spirit of this strategy, which are specific to 
the findings we've outlined in this report. 


Recommendations related to the contents of the reliable 
sources guidelines 


1. Provide funding and resources to redevelop the guidelines with a task force of a 
broader range of stakeholders, including trainers, librarians, and academic and 
community-based subject matter experts. The task force can revitalize the guidelines 
in the following ways: 


a. De-center English Wikipedia's definition of reliable source and Western- 
centric biases. 


b. Improve the guidelines in each language foregrounding references to 
scholarship in the social sciences and humanities that addresses the historical 
and cultural specificities of the concept of "reliability." 


c. Offer specific guidelines for editors on how to address ways scholarship 
and news media can reproduce biases and marginalizations through claims to 
objectivity. In addition, the task force can guide editors to evaluate institutional 
materials such as catalogs, databases, indexes, bibliographies, pamphlets, 
online magazines or press releases as reliable sources of factual information. 


83 "Wikimedia 2030 Movement Strategy Recommendations.” 
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d. Provide guidance for editors to use alternative methods to assess source 
reliability.Forinstance,duringtheTownHallaboutSpanishWikipedia, participants 
observed that the challenge posed by the scarcity and dispersion of reliable 
sources of information on women and other marginalized communities can be 
overcome through the use of the Triangulation method, *^ which has been taken 
up by feminist researchers to reveal new information.** Participants expressed 
that Reliable Source (and Notability) guidelines on Wikipedia would benefit 
from the contrasted and complimentary use of different types of information 
sources (e.g., primary and secondary sources; sources that address the topic 
directly or tangentially). This would generate a more receptive environment to 
the presence of content on marginalized topics and communities, as the use of 
the same set of rules to determine the reliability of sources without taking into 
account the partial and situated character of knowledge results in the rejection 
of identities that do not fit the standard. 


e. Offer guidance to editors on how to approach sources where there is a 
presence of hateful or harmful speech, or ambiguity about fact-checking. 


Recommendations related to community processes 


2. Revisit the user consensus definition and processes. We have called into question 
the problem of relying on "silence" as consensus. We suggest that a task force is 
assembled to develop a user consensus process that is welcoming to all editors and 
potential editors. 

3. Celebrate and uplift the work of trainers and Wikipedians-in-residence. There needs 
to be more direct support for trainers positions, especially given how consensus works 
on Wikipedia. 

4. Enable the VisualEditor across all platforms including Talk pages and Meta Wikimedia 
to facilitate engagement, as well as provide support and promotion to new features 
such as Tools for Discussions, which is currently in beta. This set of tools for Talk pages 
aims to make participation easier through improvements like creating Reply to and 
New discussion buttons.?* 


84 Triangulation refers to using more than one particular approach when doing quantitative or 
qualitative research to get richer, fuller data and/or to help confirm the results of the research. 
Wilson, "Research Methods: Triangulation." However, the concept of triangulation has been acquiring 
a wide range of meanings and uses in the social sciences, from a tool of convergence with the aim of 
validation, especially of quantitative data by qualitative information, to one that seeks divergence to 
add complexity and richness to the understanding of phenomenons. Hesse-Biber, "Feminist 
Approaches to Triangulation: Uncovering Subjugated Knowledge and Fostering Social Change in 
Mixed Methods Research." 

Hesse-Biber has described how feminist researchers have extended the concept of triangulation as 
an analytical tool to shed light on subjugated knowledge by confronting quantitative and qualitative 
information to find dissonances or complementarities that reveal new information that can impact 
social change for marginalized communities. Hesse Biber, "Feminist Approaches to Triangulation: 
Uncovering Subjugated Knowledge and Fostering Social Change in Mixed Methods Research." 
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CONCLUSION 


We began this research by asking what the guidelines are on reliable sources in the 

English, French and Spanish Wikipedias. How is a "reliable source" defined in practice? 
«eX What is the impact of these guidelines on content about marginalized communities in 
d REA these three Wikipedias? To address these questions, we leveraged Art+Feminism’s 
a network of hundreds of information activists who have participated in the Wikimedia 
movement in an intersectional qualitative research project. Our study combines 
interpretative analysis of the guidelines in situ with reported experiences gleaned 
from three Town Hall community conversations about the reliable sources guidelines 
and editing. Our findings argue that the guidelines themselves are unreliable. We 
showcase how contextualized knowledges that matter to marginalized communities 
are subordinated or challenged by the definitions of “reliable source,” due to the lack 
of rigor in the guidelines and their translations. 


se $e Pea Ultimately, the Reliable Our analysis showcases three ways that the guidelines impact 
E Source guidelines are — the inclusion of content for marginalized communities. There are 
few citations, dozens of hyperlinks, and no reviews of academic 
sources on the historical and cultural meaning of reliability. The 
guidelines do not mention how scholarship and news media 
have reproduced biases, marginalizations, and erasures through 
claimstoobjectivity. Noristhere mention of the various ways thatfactuality isestablished 
in institutional settings, with metadata in catalogs, databases, indexes, bibliographies, 
pamphlets, online magazines or press releases. The English version of this guideline 
was written first. Versions of the French and Spanish guideline were created through 
translation, which upholds Western knowledge formations and the presence of English 
as a geopolitical force. Ultimately, the Reliable Source guidelines are an unreliable and 
incomplete guide to provide editors with a meaningful understanding of how to assess 
source reliability. 


an unreliable and 
incomplete guide 


The editing process of community pages is exclusionary. In the case of the articles on 

reliable sources, these were developed in all three languages by small groups of editors. 

xd We show how the rule, "if you disagree the onus is on you to say so,” is problematic. 
X EN Silence as a signal of consent in practice privileges more experienced editors or those 
who feel comfortable participating in dialogues on the platform. Wikipedia's user 
interface, specifically Media Wiki, and numerous editorial rules make it difficult for new 

and casual contributors to participate. As such, our research highlights the crucial role 

of trainers in fielding questions and providing a buffer between the platform and the 


86 The updates about this tool are published in: MediaWiki contributors, “Talk pages project/Updates." 
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During the course of the research, we became aware of the difficulties in retrieving 
evidence of discussions on the deletion of articles that were not maintained in the 
crowdsourced encyclopedia. This limited the number of case studies we were able to 
analyze. For this reason, we believe that it is essential to document the cases in which 
controversies arise regarding the interpretation of the guidelines on reliable sources 
in order to inform future research. We have concluded the study with actionable 
recommendations based on our communications and interpretative analysis. 
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In summary, guidelines on reliable sources matter to the users who engage with 
Wikipedia. They are the cornerstone to the distributed, remote collaboration between 
strangers. In calling the reliable source guidelines unreliable, we are emphasizing 
crucial pain points and weaknesses in this important aspect of the Wikipedia project, 
which is built on the importance of citations for matters of fact. Without citations, and 
with a limited consensus process, the guidelines will continue to be unreliable. We 
emphasize the important work of trainers as a way to circumvent this, but trainers 
cannot shoulder all of the inconsistencies themselves. Rather, as our recommendations 
lay out, we are hopeful that the unreliability of the guidelines is not unresolvable, rather, 
we advocate for persistence and an openness to change. 


x 
A 
Xx KO x 
KX 

x 
x 


xXx 


Xx 
* 

» 
Xx 


XXX XXX XXX 
X 
X X xXx 
Xx 
XXX Xx 


x X 
90€ EX X 
X sex X 


X» 
XX 
X QS X 


CAPACITA KA AAN 
x 
MSE KK SE ERIKA XXX € 


RIRES 
Xx 


x 
x 


x 
«d 
x 

x 
4 
«| 
x 
ZA 
«| 
x 


x X x 


X XXX X 
Xx* xx* 


AX ye 2 
tx t 


(X 
Xx ex 
WK 
x x' 
ER 

es 


Xx 
X X 


X^ ^u ye 


UNRELIABLE GUIDELINES | APPENDIX 


APPENDIX 


Figure 1.1 


| 35 


Change over time in edits and size of the English Reliable Source Guideline article page 
(2005-2020). This graph shows how a smaller number of editors were responsible for 


contributing the most, and most recent, editorial changes. 


Year counts 


900 60,000 
800 
50,000 
700 
600 40,000 
P" 500 
2 o 
ü 30000 & 
400 
300 20,000 
200 
10,000 
100 | j | 
La [| la E L m L L L L L a L L i 0 
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 
Source 
View History 


https://xtools.wmflabs.org/articleinfo/en.wikipedia.org/Wikipedia:Reliable_sources#year- 


counts 


UNRELIABLE GUIDELINES | APPENDIX | 36 


Figure 1.2 


Change over time in edits and size of the French Reliable Source Guideline article page 
(2005-2021). This graph shows how a smaller number of editors were responsible for 
contributing the most, and most recent, editorial changes on French Wikipedia. 


The text is a translation of the English guideline. 
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Figure 1.3 


Change over time in edits and size of the Spanish Reliable Source Policy Guideline 
article page (2008-2021). The graph shows that nearly 5096 of the content (8,870 
bytes) was developed in 2008, with only 4 edits. The page almost doubled in size 
(15,646 bytes) in 2009, and so increased the number of edits (336). 


The content's size hit a stable point in 2013 (14,464 bytes) and has not increased 
since then. 
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GLOSSARY 


We choose not to use Wikipedia (WP) shorthand in the document. However we 
recognize the ways in which unfamiliarity with these terms can cause further barriers 
for participation. Below is a list of commonly used shortcuts and their definitions. 


WP - Shorthand for Wikipedia 


Enwiki - English Wikipedia 


WP:AIC - Articles for Creation 

WP:AID - Articles for Deletion 

WP:GNG - General Reliability Guidelines 
WP:NFP - General principles 

WP:NOR - No original research 

WP:NPOV and WP: PVN - Neutral Viewpoint 
WP:OBS - Obscurity 

WP:RS - Reliable Sources 

WP:V - Verifiability 

WP:VER - Verifiability 


Frwiki - French Wikipedia 


WP:FIABLE - Sources Fiables 

WP:CVS - Citez vos sources 

WP:SOURCE - Projet source 

WP:ODS and WP:OBS - Observatoire des sources 


Eswiki - Spanish wikipedia 


WP:FF - Fuentes fiables 

WP:VER - Verificabilidad 

WP:PVN - Punto de vista neutral 
WP:NO- Lo que Wikipedia no es 
WP:BPV - Biografías de personas vivas 


WME - Wikimedia Foundation 


Offwiki - In person, or otherwise outside of Wikipedia 
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Attribution — You must give appropriate credit, provide a link to the license, 
and indicate if changes were made. You may do so in any reasonable manner, 
but not in any way that suggests the licensor endorses you or your use. 


NonCommercial — You may not use the material for commercial purposes. 
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No additional restrictions — You may not apply legal terms or technological 
measures that legally restrict others from doing anything the license permits. 
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